Describing Spoken Dialogue Systems Differences

Conventional metrics in evaluating task-oriented Spoken Dialogue Systems (SDSs) are objective metrics such as the mean number of turns, or the estimated success rate. We provide further empirical evidence that these metrics are unreliable to distinguish across SDS: they fail to distinguish changes in the system, and detect differences where there are none. For instance, we report that the mean estimated success can have statistically significant differences (at the 5% significance level) on exactly the same system, over different, contiguous periods of time. We propose Dialogue System Difference Finder (DSDF), a novel model which can explain that the differences found using conventional metrics are due to seasonal and usage characteristics. DSDF is able to describe differences between multiple SDS, and it is different from traditional performance metrics used in the SDS community, in that it is more sensitive to changes among systems, has lower variance, and can also explain what the differences consist of. We used this model to find unexpected changes in our data sets. We believe that this lays the groundwork toward building a fully-automatic metric.